Skip to content

System tests: Add configurable simulation timeout per test case#735

Merged
MakisH merged 7 commits into
precice:developfrom
AdityaGupta716:systemtests/add-timeout-support
May 12, 2026
Merged

System tests: Add configurable simulation timeout per test case#735
MakisH merged 7 commits into
precice:developfrom
AdityaGupta716:systemtests/add-timeout-support

Conversation

@AdityaGupta716
Copy link
Copy Markdown
Contributor

@AdityaGupta716 AdityaGupta716 commented Mar 2, 2026 •
edited by MakisH
Loading

Adds an optional timeout field to entries in tests.yaml, allowing per-test timeout configuration instead of a single hardcoded global value.

Closes #371

Changes

  • systemtests/TestSuite.py — parses optional timeout from tests.yaml; validates that the value is positive
  • systemtests/Systemtest.py — solver run and fieldcompare now use the per-test configured timeout; sets the default timeout to 900
  • tools/tests/README.md — documented the new optional field with an example

Example usage in tests.yaml

- path: heat-exchanger
  case_combination:
    - fluid-openfoam
    - solid-calculix
  reference_result: ./heat-exchanger/reference-results/...
  timeout: 3600

Backward compatible — existing entries without timeout use the previous default of 1200.

Checklist

  • I added a summary of any user-facing changes in changelog-entries/735.md
  • I will remember to squash-and-merge

@AdityaGupta716 AdityaGupta716 force-pushed the systemtests/add-timeout-support branch from 51e92a0 to 1ffd4f0 Compare March 2, 2026 10:23
@AdityaGupta716
Copy link
Copy Markdown
Contributor Author

AdityaGupta716 commented Mar 2, 2026

Hi @MakisH, the timeout is currently attached to ReferenceResult since that's where per-test configuration lives. Happy to move it to a dedicated field or separate dataclass if you prefer a different structure. Also kept BUILD_TIMEOUT fixed since build time feels infrastructure-dependent rather than simulation-dependent — let me know if you'd like that configurable too.

@MakisH MakisH added GSoC Contributed in the context of the Google Summer of Code systemtests labels Mar 6, 2026
@precice-bot
Copy link
Copy Markdown
Collaborator

precice-bot commented Mar 8, 2026

This pull request has been mentioned on preCICE Forum on Discourse. There might be relevant details there:

https://precice.discourse.group/t/gsoc-2026-aditya-gupta/2773/4

MakisH
MakisH reviewed Mar 26, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@MakisH MakisH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! This is going in a good direction, but needs some further refactoring and cleanup. See some comments.

Comment thread tools/tests/metadata_parser/metdata.py
Comment thread tools/tests/systemtests/Systemtest.py Outdated
Comment thread tools/tests/components.yaml
Comment thread tools/tests/README.md Outdated
Comment thread changelog-entries/735.md Outdated
Comment thread tools/tests/systemtests/TestSuite.py Outdated
Comment thread tools/tests/systemtests/Systemtest.py Outdated
Comment thread tools/tests/systemtests.py Outdated
@MakisH MakisH mentioned this pull request Mar 26, 2026
Merged
2 tasks
@AdityaGupta716 AdityaGupta716 mentioned this pull request Mar 26, 2026
Merged
2 tasks
@AdityaGupta716 AdityaGupta716 force-pushed the systemtests/add-timeout-support branch from e4b349c to d7345a4 Compare March 26, 2026 23:31
MakisH
MakisH reviewed May 12, 2026
View reviewed changes
Comment thread flow-over-heated-plate/reference-results/fluid-openfoam_solid-fenics.tar.gz
@MakisH MakisH changed the title systemtests: add configurable timeout per test case System tests: Add configurable timeout per test case May 12, 2026
@MakisH MakisH changed the title System tests: Add configurable timeout per test case System tests: Add configurable simulation timeout per test case May 12, 2026
@MakisH MakisH requested a review from Copilot May 12, 2026 20:00
Copilot AI reviewed May 12, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for configuring a per-test-case timeout in system tests, intended to replace relying on a single global timeout value.

Changes:

  • Adds a timeouts collection to TestSuite and plumbs a per-test timeout through the CLI runners into Systemtest.
  • Updates Systemtest to use the per-test timeout for solver run and field-compare phases.
  • Documents the new YAML field and adds a changelog entry.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
tools/tests/systemtests/TestSuite.py Stores per-test timeout values from tests.yaml alongside existing per-case settings.
tools/tests/systemtests/Systemtest.py Uses self.timeout for docker compose up and field-compare timeouts; changes how GLOBAL_TIMEOUT is defined.
tools/tests/systemtests.py Passes per-test timeout into Systemtest instances.
tools/tests/generate_reference_results.py Passes per-test timeout into Systemtest during reference generation.
tools/tests/README.md Documents the timeout field and defaulting behavior.
changelog-entries/735.md Adds a user-facing changelog note about the new timeout field.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tools/tests/systemtests/TestSuite.py Outdated
Comment thread tools/tests/systemtests.py
Comment thread tools/tests/generate_reference_results.py
Comment thread tools/tests/systemtests/Systemtest.py
Comment thread tools/tests/README.md
Comment thread changelog-entries/735.md
@MakisH
Copy link
Copy Markdown
Member

MakisH commented May 12, 2026 •
edited
Loading

@copilot apply changes based on the comments in this thread (fails for reasons related to our Git LFS config)

MakisH
MakisH approved these changes May 12, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@MakisH MakisH left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this locally, and it works as expected.

@MakisH MakisH merged commit 4d80037 into precice:develop May 12, 2026
1 check passed
@AdityaGupta716 AdityaGupta716 deleted the systemtests/add-timeout-support branch May 13, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@MakisH MakisH MakisH approved these changes

Assignees

No one assigned

Labels

GSoC Contributed in the context of the Google Summer of Code systemtests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Timeout to kill the simulation when it "hangs"

4 participants

@AdityaGupta716 @precice-bot @MakisH